Image pickup apparatus that continuously takes images to obtain multiple images, control method therefor, and storage medium

ABSTRACT

An image pickup apparatus capable of selecting an image more suited to a user from images that have been continuously taken. A face included in each of a plurality of images is chosen, and with respect to each of the images, an evaluation value for use in selecting one image from the plurality of images is calculated based on the face chosen in each of the images. An image with the highest evaluation value is selected, and when there are a plurality of images with the highest evaluation value, an image taken first, is selected. The selected image is then recorded.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image pickup apparatus, a controlmethod, and a storage medium, and more particularly to an image pickupapparatus that continuously takes images to obtain multiple images, acontrol method therefor, and a storage medium storing a program forimplementing the method.

2. Description of the Related Art

Conventionally, image processing techniques that enable images having aface with an expression desired to be remained is obtained from amongcontinuously-taken images.

For example, face images are detected from images, and facial expressionevaluation values corresponding to respective facial expressions arecalculated from the detected face images, and facial expressions aredetermined based on the relationship between the facial evaluationvalues and threshold values provided for the facial evaluation, values(see, for example, Japanese Laid-Open Patent Publication (Kokai) No.2009-253848). Also, there has been proposed a method that an imagepickup means is provided for continuously obtaining a plurality ofimages, and the images are ranked using at least one of the followingconditions, a state of a main subject, image composition, and a state ofhuman faces as a predetermined condition (see, for example, JapaneseLaid-Open Patent Publication (Kokai) No. 2009-272740).

However, according to the invention described in Japanese Laid-OpenPatent Publication (Kokai) No. 2009-253848, because threshold values areused for determination, facial expressions are determined in only secondstages.

Also, the invention described in Japanese Laid-Open Patent Publication(Kokai) No. 2009-272740 has the problem that the start timing of imagepickup which is information reflecting the intent of a user is not takeninto consideration although states of main subjects are taken intoconsideration. Specifically, even when each of a plurality of imagesthat have been continuously taken are evaluated, and a highest rankedimage is selected, this does not always suit to a user. For example,even when an image taken immediately after a release button for startingimage pickup is pressed by a user is given a lower evaluation than animage taken several images later, there is only a small differencebetween them. In this case, the difference does not raise a problem forthe user, and it can be thought that an image taken near the time atwhich the release button is intentionally operated by the user is likelyto suit the user.

Thus, the conventional arts have the problem that an image selected fromcontinuously-taken images is not always an image suited to a user.

SUMMARY OF THE INVENTION

The present invention provides an image pickup apparatus capable ofselecting an image more suited to a user can be selected from imagesthat have been continuously taken, a control method for the image pickupapparatus, and a computer-readable storage medium storing a program forimplementing the method.

Accordingly, a first aspect of the present invention provides an imagepickup apparatus that continuously takes images of a subject to obtain aplurality of images representing the subject, comprising a choosing unitconfigured to choose a face included in each of the plurality of images,a calculation unit configured to calculate an evaluation value, which isused to select one image from the plurality of images, based on the facechosen in each of the plurality of images by the choosing unit, withrespect to each of the plurality of images, a selection unit configuredto select an image with the highest evaluation value calculated by thecalculation unit, and when there are a plurality of images with thehighest evaluation value calculated by the calculation unit, select animage taken first, and a recording unit configured to record the imageselected by the selection unit.

Accordingly, a second aspect of the present invention provides a controlmethod for an image pickup apparatus that continuously takes images of asubject to obtain a plurality of images representing the subject,comprising a choosing step of choosing a face included in each of theplurality of images, a calculation step of calculating an evaluationvalue, which is used to select one image from the plurality of images,based on the face chosen in each of the plurality of images in thechoosing step, with respect to each of the plurality of images, aselection step of selecting an image with the highest evaluation valuecalculated in the calculation step, and when there are a plurality ofimages with the highest evaluation value calculated in the calculationstep, selecting an image taken first, and a recording step of recordingthe image selected in the selection step.

Accordingly, a third aspect of the present invention provides acomputer-readable non-transitory storage medium storing a program forcausing a computer to implement a control method for an image pickupapparatus that continuously takes images of a subject to obtain aplurality of images representing the subject, the control methodcomprising a choosing step of choosing a face included in each of theplurality of images, a calculation step of calculating an evaluationvalue, which is used to select one image from the plurality of images,based on the face chosen in each of the plurality of images in thechoosing step, with respect to each of the plurality of images, aselection step of selecting an image with the highest evaluation valuecalculated in the calculation step, and when there are a plurality ofimages with the highest evaluation value calculated in the calculationstep, selecting an image taken first, and a recording step of recordingthe image selected in the selection step.

According to the present invention, an image more suited to the user canbe selected from images that have been continuously taken.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF TEH DRAWINGS

FIG. 1 is a block diagram schematically showing an exemplary arrangementof ah image pickup apparatus according to an embodiment of the presentinvention.

FIG. 2 is a flowchart showing an image data recording process carriedout by a system controller appearing in FIG. 1.

FIG. 3 is a flowchart showing the procedure of a face evaluationcalculation process carried out in step S202 in FIG. 2.

FIGS. 4A and 4B are flowcharts showing the procedure of a smileevaluation process carried out in step S302 in FIG. 3.

FIGS. 5A and 5B are diagrams useful in explaining smile evaluationvalues assigned in the process in FIGS. 4A and 4B, in which FIG. 5Ashows smile level, and FIG. 5B shows smile evaluation value.

FIGS. 6A and 6B are flowcharts showing the procedure of an eye openevaluation process carried out in step S303 in FIG. 3.

FIGS. 7A and 7B are flowcharts showing the procedure of an eye openchange detection process carried out in step S507 in FIG. 6A.

FIGS. 8A to 8C are diagrams useful in explaining an eye open evaluationvalue assigned in the process in FIGS. 6A and 6B, in which FIG. 3A showseye open level in the right eye, FIG. 8B shows eye open level in theleft eye, and FIG. 8C shows eye open evaluation value.

FIG. 9 is a diagram useful in explaining a face evaluation valuecalculated in the process in FIG. 2.

DESCRIPTION OF THE EMBODIMENTS

The present invention will now be described in detail with reference tothe drawings showing an embodiment thereof.

FIG. 1 is a block diagram schematically showing an arrangement of animage pickup apparatus according to an embodiment of the presentinvention.

Referring to FIG. 1, a digital camera 100 which is the image pickupapparatus according to the present embodiment includes a systemcontroller 108, an image pickup lens 101, an image pickup device 102, anA/D converter 103, an image processing unit 104, and a format converter105. The digital camera 100 further includes a DRAM 106, an imagerecording unit 107, and a face detection module 114. The digital camera100 further includes a smile detection module 115, an eye regiondetection module 116, an eye open detection module 117, a console 109,an image pickup mode SW 110, a main SW 111, and SWs 112 and 113.

The system controller 108 controls the entire digital camera 100, andcontrols processes such as an image pickup sequence. The image pickuplens 101 includes a zoom mechanism, a focus lens mechanism, and adiaphragm shutter mechanism. The image pickup device 102 is alight-receiving means and a photoelectric conversion means forconverting reflected light, from a subject into an electric signal. TheA/D converter 103 includes a CDS circuit that, removes output noise fromthe image pickup device 102, and a nonlinear-amplification circuit thatcarries out nonlinear amplification before A/D conversion. The imageprocessing unit 104 extracts signal components in a specific frequencyband related to brightness from a signal output from the A/D converter103, and detects a focusing state of the subject. The format converter105 converts a signal output, from the image processing unit 104 intodigital image data (hereafter referred to merely as an “image”). A DRAM106 is a high-speed built-in memory (for example, a random accessmemory) in which an image output from the format converter 105 isrecorded. The DRAM 106 is used as a high-speed buffer which is atemporary image storage means, or as a work memory for image compressionand expansion. The image recording unit 107 includes a recording mediumsuch as a memory card and its interface.

The console 109 is for a user to operate the digital camera 100. Theconsole 109 includes a menu switch for configuring various settings ofthe digital camera 100 such as image pickup functions and imageregeneration settings, a zoom lever for instructing the image pickuplens to perform a zooming operation, an operation mode switch forswitching between an image pickup mode and a reproduction mode.

The image pickup mode SW 110 is a switch for configuring settings suchas determination as to whether or not to carry out face detection. Themain SW 111 is a switch for turning on the power to the digital camera100. The SW 112 is a switch for taking image pickup standby actions suchas AF (auto focus) and AE (auto exposure). The SW 113 is an image pickupswitch for taking images after the SW 112 is operated. The SW 112 andthe SW 113 are usually comprised of one button, and when the button ispressed halfway down, the SW 112 is turned on, and when the button isfurther pressed all the way down, the SW 113 is turned on.

The face detection module 114 carries out face detection using an imagesignal processed by the image processing unit 104, and sends detectedone or more pieces of face information (for example, position, size, andreliability) to the system controller 108. The face detection module 114carries out a well-known face detection process on an image output fromthe image processing unit 104, and detects a face region of a personincluded in an image taken by the image pickup device 102. It should benoted that examples of the well-known face detection process include amethod that a skin tone region is extracted from tone colors ofrespective pixels of an image, and a face is detected based on thedegree of matching with a face contour plate prepared in advance.Moreover, there has been disclosed a method that a face is detected byextracting feature points of a face such as eyes, nose, and mouth usinga known pattern recognition technique.

The smile detection module 115 calculates a face smile level for theface detected by the face detection module 114. Specifically, the smiledetection module 115 obtains feature amounts required to calculate thedegree of smiling such as contours of face constituents which constitutea face and include eyes, nose and mouth, and positions of the faceconstituents such as inner corners of eyes, outer corners of eyes,nostrils, corner of mouth, and lip. Examples of the method to obtainfeature amounts include a method using template matching based ontemplates of respective face constituents, and a method using adetermination unit for each face constituent obtained by machinelearning using a number of sample images of face constituents. The smiledetection module 115 calculates a smile level indicative of the degreeto which a face smiles based on the above described feature amounts.

The eye region detection module 116 detects an eye region from the faceregion detected by the face detection module 114.

The eye open detection module 117 detects the area of a brightnessregion corresponding to a black eye from a histogram in the eye regiondetected by the eye region detection module 116. Then, the eye opendetection module 117 calculates an eye open level, which is indicativeof the degree to which the eyes open, according to the area of a regioncorresponding to a black eye relative to an eye region.

It should be noted that methods for face detection, smile levelcalculation, and eye open level calculation are not limited to thosedescribed above, but various well-known methods may be used.

With the arrangement described above, in the present embodiment, whenthe user operates the SW 113, the system controller 108 startscontinuously taking images. Then, system controller 108 records theobtained images on the DRAM 106, selects one with the highest evaluationvalue, to be described later, from among them, and records the selectedimage in the image recording unit 107 which is a nonvolatile memory. Itshould be noted that when there are a plurality of pieces of image datawith the highest evaluation value, an image taken at a time closest tothe time at which the SW 113 was operated, that is, an image obtainedfirst is selected from the plurality of pieces of image data. It shouldbe noted that, in the following description, continuously taking imagesmay be referred to as “continuous shooting”.

FIG. 2 is a flowchart showing an image data recording process carriedout by the system controller 108 appearing in FIG. 1.

The image data recording process is a process in which a plurality ofimages indicative of a subject are obtained by continuously shooting thesubject in response to the SW 113 being turned on, and an image selectedfrom the obtained plurality of images is recorded in the image recordingunit 107.

Referring to FIG. 2, the system controller 108 carries out an imagepickup process in which it reads out an electric charge signal from theimage pickup device 102, and writes taken images on a predetermined areaof the DRAM 106 via the A/D converter 103, the image processing unit104, and the format converter 105 (step S201). The system controller 108carries out a face evaluation calculation process in FIG. 3, to bedescribed later, to evaluate the taken images (step S202). Then, thesystem controller 108 determines whether or not to update an image to berecorded on the image recording unit 107 (hereafter referred to as“to-be-recorded image data”), that is, whether or not to select thisimage (step S203) (selection unit). When it is determined in the stepS203 that the to-be-recorded image data is not to be updated (NO in thestep S203), the process proceeds to step S205 with the step S204skipped. On the other hand, when it is determined in the step S203 thatthe to-be-recorded image data is to be updated (YES in the step S203),the to-be-recorded image data is updated to the images subjected to theevaluation in the step S202 (step S204). Then, the system controller 108determines whether or not a predetermined number of images have beencontinuously taken (step S205). When it is determined that thepredetermined number of images have been continuously taken (YES in thestep S205), the system controller 108 causes the image recording unit107 to record the to-be-recorded image data recorded on the DRAM 106(step S206) (recording unit), and terminates the process.

On the other hand, when it is determined in the step S203 that theto-be-recorded image data is not to be updated (NO in the step S203),the system controller 108 proceeds the process to the step S205. When itis determined in the step S205 that the predetermined number of imageshave not been continuously taken (NO in the S205), the system controller108 repeatedly executes the step S201 and the subsequent steps.

A description will now be given of how it is determined in the step S203whether or not to update the to-be-recorded image data. One ofconditions for update is the condition A that image data subjected todetermination is “the first image taken in sequential shooting”. Theother one of the conditions for update is the condition B that “no minusevaluation value is assigned to all the faces subjected to theevaluation, and any of the faces subjected to the evaluation has a plusevaluation value for face and has a greater face evaluation value thanall other evaluation values”. When either one of the conditions A and Bis satisfied, the to-be-recorded image data is updated. In the followingdescription, an evaluation value for a face may be referred to as “faceevaluation value”.

When the face evaluation values are the same, this means that not“greater than any other evaluation values”, and hence the to-be-recordedimage data is not updated, and an image close to the time at which imagepickup is started is selected even when the face evaluation values arethe same.

According to the process in FIG. 2, when there are a plurality of imageswith the highest evaluation value, an image taken first is selected (YESin the step S203). As a result, the image that is temporary closest thetime at which image pickup is started is selected, so that the imagemore suited to the user can be selected.

A minus evaluation value means an evaluation value that is assigned to aface when the degree to which a face subjected to evaluation (hereafterreferred to as “evaluation target face”) smiles lowers a predeterminedamount or more, and further, when the degree to which the eyes openlowers a predetermined amount or more. A minus evaluation value isassigned when the evaluation target face changes from a smiling face toa normal face, and when the evaluation target face changes from an eyeopening state to an eye closing state.

A plus evaluation value means an evaluation value that is assigned to aface basically when the degree to which an evaluation target face smilesrises a predetermined amount or more, and further, when the degree towhich the eyes open rises a predetermined amount or more. A plusevaluation value is assigned when an evaluation target face changes froma normal face to a smiling face, and when an evaluation target facechanges from an eye opening state to an eye closing state.

The above-mentioned evaluation target face is a face for which a smilelevel and an eye open level are calculated to calculate an evaluationvalue among faces detected from an image. The evaluation target face isdetermined from “a face being present within a predetermined range of animage”, “a face with a predetermined size or larger size”, “a facefocused at the start of image pickup”, and so on. Namely, the evaluationtarget face is the face or faces of one or a plurality of personsintended as a subject by the user. From the second and subsequent imagestaken by continuous shooting, the same face as a face targeted forevaluation in the previous image pickup is preferentially adopted as anevaluation target face.

Whether or not the faces are the same may be determined using a facerecognition technique, or when the present evaluation target face issubstantially the same in size and position as an evaluation target facein the previous image pickup, it may be determined that the presentevaluation target, face is the same face as an evaluation target face inthe previous image pickup.

FIG. 3 is a flowchart showing the procedure of the face evaluationcalculation process carried out in the step S202 in FIG. 2.

Referring to FIG. 3, the system controller 108 detects faces from animage, and chooses evaluation target, faces from the detected facesusing the evaluation target face choosing method described above (stepS301) (selection unit). The system controller 108 carries out a smileevaluation process in FIGS. 4A and 4B, to be described later, to dosmile evaluation on any evaluation target face (step S302). The systemcontroller 108 carries out an eye open evaluation process in FIG. 6, tobe described later, to do a eye open evaluation on the same evaluationtarget face as the face on which the smile evaluation has been carriedout (step S303). The system controller 108 then calculates a faceevaluation value of the evaluation target face (step S304). In thepresent embodiment, the face evaluation value is calculated by adding upall the smile evaluation values and eye open evaluation values assignedto each evaluation target face.

The system controller 108 then determines whether or not evaluations onall the evaluation target faces within the image have been completed(step S305). Upon determining that evaluations on all the evaluationtarget faces within the image have not been completed (NO in the stepS305), the system controller 108 repeatedly carries out the processes inthe step S302 and the subsequent steps on the other evaluation targetfaces within the image. On the other hand, upon determining thatevaluations on all the evaluation target faces within the image havebeen completed (YES in the step S305), the system controller 106calculates image evaluation values (evaluation values) based on thefaces selected from the respective images in the step S301 (step S306)(calculation unit), and terminates the present process. The evaluationvalues are used to select one image from the plurality of images thathave been taken.

In the present embodiment, a face evaluation value is calculated byadding up all the evaluation values assigned to respective ones of aplurality of evaluation target faces in an image. Of course, when thereis only one evaluation target face, a face evaluation value of thisevaluation target face is an image evaluation value. When the number ofpersons is desired to be evaluated, an image evaluation value can beobtained by adding up face evaluation values of a plurality ofevaluation target faces, and when the number of persons is not desiredto be evaluated, an image evaluation value can be obtained by averagingface evaluation values of a plurality of evaluation target faces.

FIGS. 4A and 4B are flowcharts showing the procedure of the smileevaluation process carried out in the step S302 in FIG. 3.

Referring to FIGS. 4A and 4B, the system controller 108 obtains a smilelevel indicative of the degree of smiling with respect to an evaluationtarget face using the smile detection module 115 (step S401). Thesystems controller 108 determines whether or not the present image isthe first one of images that have been continuously taken (step S402).When it is determined that the present image is the first image (YES inthe step S402), the system controller 108 assigns a basic evaluationvalue to it (step S403), and terminates the present process. On theother hand, when it is determined that the present image is not thefirst image (NO in the step S402), the system controller 108 determineswhether or not the present evaluation target face is the same as theprevious evaluation target face, that is, whether or not the presentevaluation target face is the same face as an evaluation target face inthe previous image pickup (step S404). When it is determined that thefaces are not the same (NO in the step S404), the system controller 108terminates the present process without assigning any evaluation value(step 410). The reason why it is determined in the step S404 whether ornot the faces are the same is that the persons have to be the same so asto use a smile determination predetermined value because it varies amongdifferent individuals.

On the other hand, when it is determined that the faces are the same(YES the step S404), the system controller 108 calculates a differencein smile level between the present image pickup and the previous imagepickup (step S405). In the present embodiment, the smile level of animage targeted for comparison is compared to that in an image that hasbeen previously taken, so that a change in facial expression isdetected. On this occasion, the smile level of the target image may becompared to that in an image taken a predetermined time period ago, ormay be compared to that in a taken image to be recorded. Thus, smilelevels indicative of smiling degrees are obtained with respect to thesame face in respective images.

Then, the system controller 108 then calculates the ratio between thedifference and a smile determination predetermined value (firstpredetermined value) for evaluating a change in facial expression (stepS406). The system controller 108 determines whether or not the ratio issmaller than 1 and greater than −1 (that is, whether or not the absolutevalue of the ratio is smaller than 1) (step S407). When it is determinedthat the absolute value of the ratio is smaller than 1 (YES in the stepS407), the system controller 108 determines whether or not theevaluation target face has changed to a smile in the previous image forthe first time (that is, whether or not the calculated ratio has becomepositive). When it is determined that the evaluation target face has notchanged to a smile for the first time (MO in the step S408), the systemcontroller 108 terminates the present process without assigning anyevaluation value (step S410).

On the other hand, when it is determined in the step S407 that theabsolute value of the ratio is not smaller than 1 (NO in the step S407),the system controller 108 determines whether or not the ratio is equalto or greater than 1 (step S411). When it is determined that the ratiois not equal to or greater than 1 (that is, the ratio is equal to orsmaller than −1) (NO in the step S411), the system controller 108assigns a minus evaluation value (step S412) and terminates the presentprocess. The minus evaluation value assigned in the step S412 may beassigned according to the ratio obtained in the step S406. In this case,the larger the degree of change to a normal face, the greater in a minusdirection an evaluation value to be assigned.

On the other hand, when it is determined in the step S408 that theevaluation target face has changed to a smile for the first time (YES inthe step S408), the system controller 108 assigns a plus evaluationvalue (step S409) and terminates the present process. The plusevaluation value assigned in this step S409 after the step S406 is auniform value.

When it is determined that the ratio is equal to or greater than 1 (YESin the step S411), the system controller 108 assigns a plus evaluationvalue (step S409) and terminates the present process. The plusevaluation value assigned in this step S409 after the step S411 may beassigned according to the ratio obtained in the step S406. In this case,the larger the degree of change to a smile, the greater in a plusdirection an evaluation value to be assigned.

When in the process in FIG. 4B, it is determined that the evaluationtarget face has changed to a smile for the first, time with respect tothe previous image (YES in the step S407 and YES in the step S408), thesystem controller 108 assigns a plus evaluation value only once evenwhen the degree of the change is small. In an image in which a facialexpression has just changed to a state determined as being a smile, thefacial expression is likely to have reached a satisfactory smile level.Thus, by assigning a plus evaluation value only once even when thedegree of smiling has not changed the smile determination predeterminedvalue or more (step S409), a next, image that has changed is likely tobe selected.

FIGS. 5A and 5B are diagrams useful in explaining smile evaluationvalues assigned in the process in FIGS. 4A and 4B, in which FIG. 5Ashows smile level, and FIG. 5B shows smile evaluation value.

In the graph of FIG. 5A, the vertical axis represents smile level, andthe horizontal axis represents the number of images that arecontinuously taken. In this graph, a white circle 601 indicates smilelevel in the first image. A difference 602 is a difference between smilelevel in the first image and a smile level in the second image. A linesegment 603 is for comparing a smile determination predetermined valueand a difference with each other, and the smile determinationpredetermined value is a segment between the nearest black circles. Thewhite circle 601, the difference 602, and the line segment 603 are usedin the same sense with respect to images other than the first one.

Particularly in FIG. 5A, as indicated by the line segment 603, adifference between the first, image and the second image is twice thesmile determination predetermined value, and in this case, the ratio is“+2”.

A difference between the second image and the third image and adifference between the third image and the fourth image are smaller thanthe smile determination predetermined value, and a difference betweenthe fourth image and the fifth image is a little greater than the smiledetermination predetermined value.

On the other hand, in the graph of FIG. 5B, the vertical axis representssmile evaluation value, and the horizontal axis represents the number ofimages that are continuously taken.

Referring to the graph of FIG. 5B, because there is no image prior tothe first image, no difference in smile level can be calculated, andhence the above described basic evaluation value such as “10” isassigned. Then, when the first image and the second image shown in FIG.5A are compared in smile level with each other, a difference betweenthem is a plus value greater than the smile determination predeterminedvalue.

Because the ratio between the difference and the smile determinationpredetermined value is “+2” as described above, the system controller103 assigns an evaluation value of “+20”, and thus the total evaluationvalue is “30”. When the second image and the third image are compared insmile level with each other, a difference between them is smaller thanthe smile determination predetermined value, and the system controller108 thus determines that the face has not changed to a smile. However,when it is determined that the face has changed to a smile for the firsttime in the previous image, and facial expression has not changed, thesystem controller 108 assigns a uniform evaluation value only once.Here, the system controller 108 assigns an evaluation value of “+10”,and thus the total evaluation value is “40”.

Next, when the third image and the fourth image are compared in smilelevel with each other, a difference between them is smaller than thesmile determination predetermined value, and the system controller 108thus determines that the face has not changed to a smile. As a result,no evaluation is assigned, and hence the total evaluation value is still“40”. Then, when the fourth image and the fifth image are compared insmile level with each other, a difference between them is a minus valuegreater than the smile determination predetermined value. Because theratio between the difference and the smile determination predeterminedvalue is not less than 1 and less than 2, an evaluation value of “−10”is assigned, and thus the total evaluation value is “30”.

FIGS. 6A and 6B are flowcharts showing the procedure of the eye openevaluation process carried out in the step S303 in FIG. 3.

Referring to FIGS. 6A and 6B, with respect to an evaluation target face,the system controller 108 causes the eye open detection module 117 toobtain an eye open level indicative of the degree to which the eyes open(step S501). The system controller 108 determines whether or not thepresent evaluated image is the first one of images that are continuouslytaken (step S502). Upon determining that the present evaluated image isthe first one of images that are continuously taken (YES in the stepS502), the system controller 108 assigns the basic evaluation value(step S503) and terminates the present process. On the other hand, upondetermining that, the present evaluated image is not the first one ofimages that are continuously taken (NO in the step S502), the systemcontroller 108 determines whether or not the present evaluation targetface is the same as an evaluation target face in the previous imagepickup (step S504). The reason why it is determined whether or not thefaces are the same is that the persons have to foe the same so as to usean eye open change determination predetermined value (secondpredetermined value) for evaluating changes in the degree to which theeyes open because it varies among different individuals. Thus, eye openlevels indicative of the degrees to which the eyes open are obtainedwith respect to the same faces in the respective images.

The eye open change determination predetermined value may be prepared inadvance as a predetermined value, or calculated from a measurement valueobtained by measuring transitions in the eye open level of each eye ofevaluation target faces obtained before the start of image pickup. Also,the eye open change determination predetermined value may be differentbetween right, and left eyes.

When it is determined in the step S504 that the present evaluationtarget face is the same as an evaluation target face in the previousimage pickup (YES in the step S504), the system controller 108determines whether or not the face has changed to a smile (step S505).

On the other hand, when it is determined in the step S504 that thepresent evaluation target face is not the same as an evaluation targetface in the previous image pickup (NO in the step S504), the systemcontroller 108 terminates the present process without assigning anyevaluation value (step S506). The reason why no evaluation value isassigned when the face has changed to a smile is that the face havingchanged to a smile is likely to close the eyes.

When it is determined in the step S504 that the face has not changed toa smile (NO in the step S505), the system controller 108 carries out aneye open change detection process in FIGS. 7A and 7B, to be describedlater (step S507). The eye open change detection process is a process inwhich a change in eye open is detected by calculating a difference inthe eye open level of each eye between the present, image pickup and theprevious image pickup. Regarding the previous image pickup subjected tocomparison in the step S507, the system controller 108 compares eye openlevels between an image in the present image pickup and an image theprevious image pickup to detect a change in facial expression. On thisoccasion, the eye open level of an image in the present image pickup maybe compared to the eye open level of an image taken a predetermined timeago, or may be compared to the eye open level of a taken image which isan image to be recorded.

Then, the system controller 108 determines whether or not the eve openlevels of both eyes have changed (step S508). When, it is determinedthat the eye open levels of both eyes have changed (YES in the stepS508), the system controller 108 determines whether or not both eyeshave changed in the same direction (step S512). When it is determinedthat both eyes have changed in the same direction (YES in the stepS512), the system controller 108 determines whether or not both eyeshave changed in an eye opening direction (step S514). When it isdetermined that both eyes have changed in an eye opening direction (YESin the step S514), the system controller 108 assigns a plus evaluationvalue (step S515) and terminates the present process. The plusevaluation value assigned here is a value according to the ratio of aneye whose ratio between the difference and the eye open changedetermination predetermined value is higher.

When it is determined in the step S508 that the eye open levels of botheyes have not changed (MO in the step S508), the system controller 108determines whether or not the eye open level of only one eye has changed(step S509). When it is determined that the eye open level of only oneeye has not changed, that is, the eye open level of neither of the eyeshas changed (NO in the step S509), the system controller 108 terminatesthe present process without assigning any evaluation value (step S506).

When it is determined that the eye open level of only one eye haschanged (YES in the step S509), the system controller 108 determineswhether or not the eye has change in an eye opening direction (stepS510). When it is determined that the eye has changed in an eye openingdirection (YES in the step S510), the system controller 108 assigns aplus evaluation value (step S511) and terminates the present process.

When it is determined in the step S510 that the eye has not changed inan eye opening direction (NO in the step S510), the system controller108 assigns a minus evaluation value (step S513) and terminates thepresent process. When it is determined in the step S512 that, both eyeshave not changed in the same direction (NO in the step S512), the systemcontroller 108 assigns a minus evaluation value (step S513) andterminates the present process. When it is determined that the eyes havenot changed in an eye opening direction (NO in the step S514), thesystem controller 108 assigns a minus evaluation value (step S513) andterminates the present process.

The minus evaluation value assigned in the step S513 is a valueaccording to the ratio of an eye whose ratio between the difference andthe eye open change determination predetermined value is higher.

The reason why a minus evaluation value is assigned when it isdetermined in the step S512 that the eyes have not changed in the samedirection is that when the facial expression is likely to be unbalancedbecause one eye has changed in an eye opening direction, and the othereye has changed in an eye closing direction.

FIGS. 7A and 7B are flowcharts showing the procedure of the eye openchange detection process carried out in the step S507 in FIG. 6A.

According to the flowcharts of FIGS. 7A and 7B, the process issequentially carried out for the right eye and the left eye in thisorder.

FIGS. 7A and 7B, the system controller 108 calculates a difference inthe eye open level of each eye between the present image pickup and theprevious image pickup (step S701). The system controller 108 thencalculates the ratio between the difference of each eye and the eye openchange determination predetermined value (step S702), and determineswhether or not the ratio of the right eye is greater than −1 and smallerthan 1 (that is, whether or not the absolute value of the ratio issmaller than 1) (step S703). When it is determined that, the absolutevalue of the ratio of the right, eye is smaller than 1 (YES in the stepS703), this means that the right eye is unchanged, and thus the systemcontroller 106 stores in the DRAM 106 that the right eye is unchanged(step S704) and proceeds to step S705.

When it is determined in the step S703 that the absolute value of theratio of the right eye is not smaller than 1 (NO in the step S703), thesystem controller 108 determines whether or hot the ratio of the right,eye is equal to or greater than 1 (step S707). When it is determinedthat the ratio of the right eye is equal to or greater than 1 (YES inthe step S707), this means that the right eye has changed in an eyeopening direction, and thus the system controller 108 stores in the DRAM106 that the right eye has changed in an eye opening direction (stepS708) and proceeds to the step S705.

When it is determined that the ratio of the right eye is smaller than 1,that is, the ratio of the right eye is not more than −1 (NO in the stepS707), this means that the right eye has changed in an eye closingdirection, and thus the system controller 108 stores in the DRAM 106that the right eye has changed in an eye closing direction (step S709)and proceeds to the step S705.

The system controller 108 then determines whether or not the ratio ofthe left eye is greater than −1 and smaller than 1 (that is, whether ornot the absolute value of the ratio is smaller than 1) (step S705). Whenit is determined that the absolute value of the ratio of the left eye issmaller than 1 (YES in the step S705), this means that the left eye isunchanged, and thus the system controller 108 stores in the DRAM 106that the left eye is unchanged (step S706) and terminates the presentprocess.

When it is determined in the step S705 that the absolute value of theratio of the left eye is not smaller than 1 (NO in the step S705), thesystem controller 108 determines whether or not the ratio of the left,eye is equal to or greater than 1 (step S710). When it is determinedthat the ratio of the left eye is equal to or greater than 1 (YES in thestep S710), this means that the left eye has changed in an eye openingdirection. Thus, the system controller 108 stores in the DRAM 106 thatthe left, eye has changed in an eye opening direction (step S711) andterminates the present process.

When it is determined in the step S710 that the ratio of the left eye issmaller than 1, that is, not more than −1 (NO in the step S710), thismeans that, the left eye has changed in an eye closing direction. Thus,the system controller 108 stores in the DRAM 106 that the left eye haschanged in an eye closing direction (step S712) and terminates thepresent process.

FIGS. 8A to 8C are diagrams useful in explaining an eye open evaluationvalue assigned in the process in FIGS. 6A and 6B, in which FIG. 8A showsthe eye open level of the right eye, FIG. 8B shows the eye open level ofthe left eye, and FIG. 8C shows eye open evaluation value.

In the graphs of FIGS. 8A and 8B, the vertical axis represents the eyeopen level of the right or left eye, and the horizontal axis representsthe number of images that are continuously taken. In these graphs, whitecircles 701 and 704 represent eye open levels of the right or left eyein the first, image. Differences 702 and 705 are differences in thesmile level of the right or left eye between the first image and thesecond image. Line segments 703 and 706 are for comparing an eye opendetermination predetermined value and a difference with each other withrespect to the right or left eye, and the eye open determinationpredetermined value is a segment between the nearest two black circles.The white circles 701 and 704 and the line segments 703 and 706 are usedin the same sense with respect to images other than the first one.

Referring to FIG. 8A, an eye open level corresponding to the difference702 between the first image and the second image is a minus value notless than the eye open determination predetermined value and less thantwice the eye open determination predetermined value, and hence thesystem controller 108 determines that the eye has changed in an eyeclosing direction. However, when it is determined that there is a smilein the second image, it is not determined whether or not the left eyehas changed in an eye closing direction. Referring to FIG. 8A, adescription will be given of an eye open evaluation value or theassumption that there is a smile in the second image.

Next, when the second image and the third image are compared in eye openlevel with each other, a difference between them is a plus value notless than the eye open determination predetermined value and less thantwice the eye open determination predetermined value, and the systemcontroller 108 thus determines that the eye has changed in an eyeopening direction. Further, when the third image and the fourth imageare compared in eye open level with each other, a difference betweenthem is a minus value not less than the eye open determinationpredetermined value and less than twice the eye open determinationpredetermined value, and the system controller 108 thus determines thatthe eye has changed in an eye closing direction.

When the fourth image and the fifth image are compared in eye open levelwith each other, a difference between them is a plus value not less thanthe eye open determination predetermined value and less than twice theeye open determination predetermined value, and the system controller108 thus determines that the eye has changed in an eye openingdirection.

Referring to FIG. 8B, an eye open level corresponding to the difference705 between the first image and the second image is a minus value notless than the eye open determination predetermined value and less thantwice the eye open determination predetermined value, and hence thesystem controller 108 determines that the eye has changed in an eyeclosing direction. However, when it is determined that there is a smilein the second image, it is not determined whether or not the eye has notchanged in an eye closing direction. Referring to FIG. 8B, a descriptionwill be given of an eye open evaluation value on the assumption thatthere is a smile in the second image.

Next, when the second image and the third image are compared in eye openlevel with each other, a difference between them is a plus value notless than the eye open determination predetermined value and less thantwice the eye open determination predetermined value, and the systemcontroller 108 thus determines that the eye has changed in an eyeopening direction. Further, when the third image and the fourth imageare compared in eye open level with each other, a difference betweenthem is less than the eye open determination predetermined value, andthe system controller 108 thus determines that the eye open level isunchanged. In addition, when the fourth image and the fifth image arecompared in eye open level with each other, a difference between them isless than the eye open determination predetermined value, and the systemcontroller 108 thus determines that the eye open level is unchanged.

In the graph of FIG. 8C, the vertical axis represents eye openevaluation value, and the horizontal axis represents the number ofimages that are continuously taken. The eye open evaluation value iscalculated based on the state of change in both eyes.

A difference in smile level cannot be calculated for the first imagebecause there is no previous image, and hence the system controller 108assigns a predetermined evaluation value, for example, “10”. In thesecond image, the right eye has changed in an eye closing direction, andthe left eye has changed in an eye closing direction, and thus thesystem controller 108 determines that the eyes have changed in the samedirection, i.e. an eye closing direction. However, when it is determinedthat there is a smile in the second image, no evaluation value for eyeopen level is assigned, and thus the evaluation value remains to be“10”. In the third image, the right eye has changed in an eye openingdirection, and the left, eye has changed in an eye opening direction,and thus the system controller 108 determines that the eyes have changedin the same direction, i.e. an eye opening direction. The ratios of boththe right and left eyes are not less than 1 and less than 2, and thus anevaluation value of “+10” is assigned, and the total, evaluation valueis “20”. In the fourth image, the right eye has changed in an eyeclosing direction, and the left eye has not changed, and thus the systemcontroller 108 determines that the eyes have changed in an eye closingdirection. The ratio of the right eye is not less than 1 and less than2, and thus an evaluation value of “−10” is assigned, and the totalevaluation value is “10”. In the fifth image, the right eye has changedin an eye opening direction, and the left, eye has not changed, and thusthe system controller 108 determines that the eyes have changed in aneye opening direction. The ratio of the right eye is not less than 1 andless than 2, and thus an evaluation value of “+10” is assigned, and thetotal evaluation value is “20”.

FIG. 9 is a diagram useful in explaining the face evaluation valuecalculated in the process in FIG. 2.

In the graph of FIG. 9, the vertical axis represents face evaluationvalue, and the horizontal axis represents the number of images that arecontinuously taken. The face evaluation value is the sum of a smileevaluation value and an eye open evaluation value.

The smile evaluation value of the first image is “10”, the smileevaluation values of the second and fifth images are “30”, and the smileevaluation values of the third and fourth images are “40”. On the otherhand, the eye open evaluation values of the third and fifth images are“20”, and the other eye open evaluation values are “10”.

Thus, in the case of smile evaluation values in FIG. 5B and eye openevaluation values in FIG. 8C, the evaluation value of the first image is“20”, the evaluation value of the second image is “40”, the evaluationvalue of the third image is “60”, the evaluation value of the fourthimage is “50”, and the evaluation value of the fifth image is “50”.

When there is only one evaluation target face, a face evaluation valuetherefor is an image evaluation value. In the present embodiment, thethird image is selected as an image to be recorded due to the conditionsthat “no minus evaluation values are assigned to all the evaluationtarget faces. Also, a plus evaluation value is assigned to any of theevaluation target faces. Also, a face evaluation value is the highest”.

It should be noted that as described above, when there are a pluralityof images that satisfy the above conditions, an image taken first isselected from the plurality of images with consideration given to thetiming in which the SW 113 is operated by the user.

Thus, when it is determined that the degree of change in the smile levelor eye open level of an evaluation target face is smaller than athreshold value, the resulting evaluation value is the same value as theprevious evaluation value. Thus, when there is only a small change infacial expression, image data can be selected with the timing ofoperation of the SW 113 by the user given higher priority than the levelof the facial expression. As a result, an image can be selected withconsideration given to both the evaluation value of the facialexpression and the timing of the operation of the SW 113 by the user,and image data more suited to the user can be stored from images thathave been continuously taken.

Although in the present embodiment, face-related degrees are the levelsof smile and eye open, evaluation values may be similarly assigned tothe composition and focusing of an image depending on the position andsize of a face and used for the selection of an image.

In the present embodiment, when there are a plurality of images with thehighest evaluation value, an image taken first is selected (YES in thestep S203). Thus, an image close to the time at which image pickup isstarted is selected, and hence an image more suited to the user can beselected.

In the embodiment described above, control of the system controller 103may he exercised by a single piece of hardware, or shared by a pluralityof pieces of hardware so as to control the entire apparatus.

Moreover, although in the embodiment described above, the present,invention is applied to the digital camera 100, the present invention isnot limited to this. Specifically, the present invention may be appliedto any display control apparatus insofar as it can provide control sothat so that, a plurality of images can be displayed at the same time.Examples of the display control apparatus include a personal computerand a PDA. Moreover, a cellular phone terminal, a portable image viewer,a display provided in a printer unit which allows selection andconfirmation of an image to be printed, and a digital photo-frame.

Other Embodiments

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment(s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2010-183297 filed Aug. 18, 2010, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image pickup apparatus that continuouslyshoots to generate a plurality of image data, comprising: a calculationunit configured to calculate, for each of the plurality of image data,an evaluation value based on a subject included in the each of theplurality of image data; a selection unit configured to select any imagedata from among the plurality of image data based on the evaluationvalue in the each of the plurality of image data and the order of theplurality of image data generated by shooting; and a storage unitconfigured to store the image data selected by the selection unit.
 2. Animage pickup apparatus according to claim 1, wherein the selection unitis configured to select an image data with the highest evaluation valuefrom among the plurality of image data, and wherein the selection unitselects, in a case where a difference between an evaluation value in animage data generated by a first shooting and an evaluation value in animage data generated by a latter shooting is a predetermined value orless, the image data generated by the first shooting from among theplurality of the image data even if the evaluation value in the imagedata generated by the latter shooting is higher than the evaluationvalue in the image data generated by the first shooting.
 3. An imagepickup apparatus according to claim 2, wherein the selection unitselects the image generated by the latter shooting in a case where theevaluation value in the image data generated by the latter shooting ishigher than the evaluation value in the image data generated by thefirst shooting and the difference between the evaluation value in theimage data generated by the first shooting and the evaluation value inthe image data generated by the latter shooting is higher than thepredetermined value.
 4. An image pickup apparatus according to claim 1,further comprises a second selection unit configured to select, from theeach of the plurality of image data, a subject based on which thecalculation unit calculates an evaluation value, wherein the calculationunit calculates, for the each of the plurality of image data, anevaluation value based on the subject selected by the second selectionunit.
 5. An image pickup apparatus according to claim 1, wherein thecalculation unit calculates a smile evaluation value for a smile of aperson included in the each of the plurality of image data.
 6. An imagepickup apparatus according to claim 1, wherein the calculation unitcalculates an eye open evaluation value for eyes of a person included inthe each of the plurality of image data.
 7. A control method for animage pickup apparatus that continuously shoots to generate a pluralityof image data, comprising: a calculation step of calculating, for eachof the plurality of image data, an evaluation value based on a subjectincluded in the each of the plurality of image data; a selection step ofselecting any image data from among the plurality of image data based onthe evaluation value in the each of the plurality of image data and theorder of the plurality of image data generated by shooting; and astorage step of storing the selected image data into a storage medium.8. A computer-readable non-transitory storage medium storing a programfor causing a computer to implement a control method for an image pickupapparatus that continuously shoots to generate a plurality of imagedata, the control method comprising: a calculation step of calculating,for each of the plurality of image data, an evaluation value based on asubject included in the each of the plurality of image data; a selectionstep of selecting any image data from among the plurality of image databased on the evaluation value in the each of the plurality of image dataand the order of the plurality of image data generated by shooting; anda storage step of storing the selected image data into a storage medium.9. An image pickup apparatus comprising: an image pickup unit configuredto continuously shoot in accordance with an instruction of a user; acalculation unit, configured to calculate, for each of a plurality ofimages obtained by the continuous shooting, an evaluation value based ona subject included in the each of the plurality of images; a selectionunit configured to select any image from among the plurality of imagesbased on the evaluation value of the each of the plurality of images andan interval between a timing of a shooting of the each of the pluralityof images and a timing of the instruction of the user; and a storageunit configured to store the image selected by the selection unit. 10.An image pickup apparatus according to claim 9, wherein the selectionunit is configured to select an image data with the highest evaluationvalue from among a plurality of image data, the plurality of image dataincluding a first image data and a second image data, and wherein theselection unit, in a case where a difference between an evaluation valuein the first image data and an evaluation value in the second image datais a predetermined value or less and a timing of a shooting which hasgenerated the first image data is closer to a timing of the instructionof the user than a timing of a shooting which has generated the secondimage data, selects the first, image data even if the evaluation valuein the second image data is higher than the evaluation value in thefirst image data.
 11. An image pickup apparatus according to claim 10,wherein the selection unit selects the second image data in a case wherethe evaluation value in the second image data is higher than theevaluation value in the first linage data and the difference between theevaluation value in the first, image data and the evaluation value inthe second image data is higher than the predetermined value.
 12. Animage pickup apparatus according to claim 9, further comprises a secondselection unit configured to select, from the each of the plurality ofimage data, a subject based on which the calculation unit calculates anevaluation value, wherein the calculation unit calculates, for the eachof the plurality of linage data, an evaluation value based on thesubject selected by the second selection unit.
 13. An image pickupapparatus according to claim 9, wherein the calculation unit calculatesa smile evaluation value for a smile of a person included in the each ofthe plurality of image data.
 14. An image pickup apparatus according toclaim 9, wherein the calculation unit calculates an eye open evaluationvalue for eyes of a person included in the each of the plurality ofimage data.
 15. A control method for an image pickup apparatus,comprising: an image pickup step of continuously shooting in accordancewith an instruction of a user; a calculation step of calculating, foreach of a plurality of images obtained by the continuous shooting, anevaluation value based on a subject included in the each of theplurality of images; a selection step of selecting any image from amongthe plurality of images based on the evaluation value of the each of theplurality of images and an interval between a timing of a shooting ofthe each of the plurality of images and a timing of the instruction ofthe user; and a storage step of storing the selected image into astorage medium.
 16. A computer-readable non-transitory storage mediumstoring a program for causing a computer to implement a control methodfor an image pickup apparatus, the control method comprising: an imagepickup step of continuously shooting in accordance with an instructionof a user; a calculation step of calculating, for each of a plurality ofimages obtained by the continuous shooting, an evaluation value based ona subject included in the each of the plurality of images; a selectionstep of selecting any image from among the plurality of images based onthe evaluation value of the each of the plurality of images and aninterval between a timing of a shooting of the each of the plurality ofimages and a timing of the instruction of the user; and a storage stepof storing the selected image into a storage medium.